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Introduction 

0.1. Let A$ be the group denned by generators x%, xs, x$ and relations x\ = x\ = 
^5 = ^23^3^5 = 1- It is well known that A$ is isomorphic to the alternating group 
in 5 letters. Let k be an algebraically closed field of characteristic p where p = 
or a prime > 7. Let G be a connected reductive algebraic group over k. We shall 
be interested in the classification of homomorphisms A5 — > G up to conjugation 
by G. When G = GL n this is the same as the classification of n-dimensional 
representations of A$ over k up to isomorphism; this can be reduced to the clas- 
sification of irreducible representations of A5 which is classical (there are five of 
them up to isomorphism, of degrees 1, 3, 3, 4, 5). In the general case, there is again 
something analogous to the notion of irreducible representation, which we call a 
regular homomorphism A5 — > G. This is, by definition, a homomorphism A5 — > G 
whose image is not contained in a Levi subgroup of a proper parabolic subgroup 
of G. Again the classification of homomorphisms A5 — > G up to conjugacy can be 
more or less reduced to the analogous question for regular homomorphisms (for G 
and also for smaller groups). 

D.D.Frey [F1],[F2],[F3] has classified up to conjugacy the homomorphisms (resp. 
non-regular homomorphisms) A$ — > G for G of type Eq, Ej (resp. Eg) over C. For 
G of type Eg he showed that there is at least one regular homomorphism A5 — > G, 
but the problem of classifying up to conjugacy the regular homomorphisms A5 — > 
G remained open. 

Serre [S] has suggested that this problem could be attacked using the complex 
representation theory of Eg(F q ), F q a finite field. He showed that the number d of 
conjugacy classes of regular homomorphisms ^5 — ► Eg(k) can be extracted from a 
certain sum over all irreducible characters of Eg(F q ), if this sum can be computed. 

The purpose of this paper is to show that this sum can be computed with enough 
precision so that a solution of the above problem is obtained. (See Theorem 4.5.) 
In fact, we show that d = 1. Thus, our results, in conjunction with the earlier 
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work of Frey, complete the classification up to G-conjugacy of homomorphisms 
A$ G for G adjoint simple. 

One of the key ingredients in our proof is a collection of inequalities (see 2.2) 
involving representations of Weyl groups. While these inequalities are conjectural 
in general, enough of them can be verified (with the aid of a computer) so that 
the proof goes through. 

I want to thank R.Griess for introducing me to this problem. I also want to 
thank J.-P.Serre for making [S] available to me and for some useful discussions. 



1. Examples 

1.1. If G is a group we denote by Zq the centre of G and by Gder the derived 
group of G; we set G a d = G/Zq. If g G G let Z G (g) be the centralizer of g in 
G and let Ca(g) be the conjugacy class of g in G. If H is a subgroup of G let 
Zq(H) (resp. Nq(H)) be the centralizer (resp. normalizer) of H in G. If G, G' 
are groups, let Hom(G, G') be the set of group homomorphisms G — > G' . 

If G is an affine algebraic group over k (as in 0.1), let G° be the identity 
component of G and let G = G/G°. 



For a finite set Y let Y or fl(Y) be the cardinal of Y . 



1.2. Assume now that G is connected, reductive algebraic group over k of simply 
laced type. Let n : G — > G be the simply connected covering of Gder- Let T{G) 
be the set of maximal tori in G. Let r(G) be the rank of G and let v{G) be half 
the number of roots of G. 

Since A$ is perfect, any homomorphism ip : A$ — > G has image contained in 
Gder- This gives us a bijection between the set of G-orbits on Hom(.4.5,G) and 
the set of G^-orbits on Hom(^4 5 , Gder)- Now, the obvious map from G-orbits on 
Hom(^4 5 ,G) to G a d-orbits on Hom(„4 5 , G^) is injective. 

(It is enough to show that, if ip, ip' G Hom^s — > G) are such that hip = hip' 
where h : G — > G a d is the obvious map, then ip = ip' . For x G A$ we have 
ip'(x) = ip(x)z(x) where z G Hom(^4 5 , Zq). Since A$ is perfect and Zq is abelian, 
we have z(x) = 1 for all x, hence ip = ip'.) 

Lemma 1.3 (Serre [S]). Let ip G Hom(^4 5 ,G). We have 
En=2,3,5 dim Z G {ip{x n )) = dim(G) + 2 dim Z G (tp(A 5 )). 

This is equivalent to its Lie algebra analogue 

dimg^™) = dim g + 2 dim g" 45 

n=2,3,5 

where g = Lie G, and the upper index denotes the fixed point set of the appropriate 
automorphism or group of automorphisms of g. More generally, this holds when 
g is replaced by a finite dimensional fc-vector space V with an ^4.5-action; we may 
assume that V is an irreducible representation of A5 (we have complete reducibility 
since ^ G k) and in that case the equality is checked by direct computation. 
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Lemma 1.4. If ip G Hom(.4.5, G), then Z G {^{A^)) is a reductive group. 

Let C(n) = C G (ip(x n )), n = 2, 3, 5. Then C(n) is an affine variety since ^{x n ) is 
a semisimple element. Consider the affine variety 

X = {(02,03,05) £ C(2) X C(3) X C(5); 2 0305 = 1}. 

The group G acts on X by conjugation on all factors. By 1.3, the dimension of 
the G-orbit of (02,03,05) G X equals 

dim(G)-( ^ dimZ G (0 n ) -dim(G))/2 = ^ dimC(n)/2; 

n=2,3,5 n=2,3,5 

in particular it is independent of the choice (02,03,0s)- Since any G-orbit of 
minimum dimension must be closed in X, it follows that any G-orbit in X is 
closed in X; hence it is affine. By a known criterion (Richardson) it follows that 
the isotropy group in G of any point of X is reductive. The lemma is proved. 

Lemma 1.5. The following four conditions for ip G Hom(^5, G) are equivalent: 

(i) dhaZ G (ip(A5)) = dimZc; 

(ii) Z G (ip(A 5 ))/Z G is finite; 

(Hi) any subtorus of Z G {ij){A^)) is contained in Z G ; 
(iv) KG)-E„ =2> 3,5^G^W))=r(M- 

This follows from Lemmas 1.3, 1.4. 

1.6. We say that ip G Hom(^4. 5 ,G) is regular if the conditions of Lemma 1.5 
are satisfied. Let Hom reg (^4 5 , G) = {ip G Hom(^l5, G); ip regular}. Note that 
ip G Hom(^4.5, G) is regular if and only if its image in Hom(^ 5 , G a d) is regular. 

We show that the classification of homomorphisms ip : A5 — > G up to G- 
conjugacy can be essentially reduced to the analogous problem for regular ho- 
momorphisms. Let ip G Hom(^45,G). Let S be a maximal torus of Z G {^>{A^)) 
and let L = Z G (S). (A Levi subgroup of a parabolic subgroup of G.) Using 
1.5(iii), we see that ip defines a regular homomorphism A5 — > L. This gives us 
a bijection between the set of G-orbits on Hom(^4 5 ,G) and the disjoint union 
over all G-conjugacy classes of Levi subgroups L of the sets of N G (L) -orbits on 
Hom refl (^4 5 ,L). 

1.7. A 235-triple for G is a triple (G2, G3, G5) of conjugacy classes in G such that, 
if g n G G n , then g™ = 1, n = 2,3, 5. The type of a 235-triple is by definition the 
sequence X 2l X 3l X 5 where X n is the type of the root system of Z G (g n ). Clearly, 
G has only finitely many 235-triples. As pointed out in [S], of particular interest 
are the 235-triples (G2, G3, G5) in G such that 

KG)- "(Z G (g n )) = r(G ad ) 

n=2,3,5 
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where g n G C n . Such 235-triples are called regular. If G Hom reg (^4 5 , G) and 
G n = CG(fp(x n )), n = 2, 3, 5 then (G 2 , G 3 , G 5 ) is a regular 235-triple. 

Let t be an automorphism of order 2 of A5 such that l{x2) is conjugate to X2, 
i(xs) is conjugate to £ 3 , l(x$) is conjugate to x|. For any ip G Hom(.4.5, G) let ''V = 
V> o t. If (G 2 , G 3 , G 5 ) is a regular 235 -triple of G, then l (G 2 , G 3 , G 5 ) = (G 2 , G 3 , G0, 
where C' 5 = {g 2 ; g G G5}, is again a regular 235 -triple of G. 

1.8. Assume that G = G a ^ is simple of type A m _\. If to G {2,3} then G 
has exactly two regular 235 -triples (interchanged by t) and there are exactly two 
ip G Hom res (^4 5 , G) up to conjugacy (interchanged by t). If m = 4, then G has 
exactly one regular 23 5-triple and there are exactly two ifj G Hom res (.4.5, G) up to 
conjugacy. If to G {5,6} then G has exactly one regular 23 5 -triple and there is 
exactly one tp G Hom reg (.4.5, G) up to conjugacy. If m > 7, then G has no regular 
23 5-triple and therefore Hom reg (^45, G) = 0. 

The type of a regular 235 -triple is (0,0,0) (if m = 2),(Ai,0,0) (if m = 3), 
(AlA u $) (if to = 4), (A 2 A l7 A2,0) (if m = 5), (if to = 6). 

For to = 3, 5 (resp. to = 2,6), ip comes from an irreducible m-dimensional 
representation of A5 (resp. of the duble cover 5"L 2 (F5) of A5, regarded as a 
projective representation of ^.5). For to = 4, one ifj comes from an irreducible 
4-dimensional representation of A5, the other i/j comes from an irreducible 4- 
dimensional representation of 5'L 2 (F 5 ) which does not factor through A5. 

1.9. Assume that G = G a d is simple of type D m . 

(a) If to = 4, then G has exactly four regular 235 -triples: two of type 
(Af,Af,$) (interchanged by t) and two of type (Af,A 2 ,$) (interchanged by t); 
correspondingly, it has exactly four G Hom reg (^45, G) up to conjugacy: two of 
them (interchanged by l) come from 8-dimensional orthogonal representations of 
A5 which decompose as 1 + 3 + 4 into irreducibles and two of them (interchanged 
by l) come from 8-dimensional orthogonal representations of A§ which decompose 
as 3 + 5 into irreducibles. 

(b) If to = 5, then G has exactly one regular 23 5-triple; it has type 
(AsAf , AiA\ , A\)\ also G has exactly two ifj G Hom reg (^4s, G) up to conjugacy; 
one of them comes from a 10-dimensional orthogonal representation of A5 which 
decomposes as 3 + 3 + 4 into irreducibles; the other comes from a 10-dimensional 
orthogonal representation of A5 which decomposes as 1+4 + 5 into irreducibles. 

(c) If to = 6, then G has exactly three regular 23 5-triples; one has type 
{A\,A^A\,A\) and the other two (interchanged by t) have type 

A 3 Af , A 2 Ai); correspondingly, it has exactly three ip G Hom reg (w45, G) up 
to conjugacy: one of them comes from a 12-dimensional orthogonal representation 
of A$ which decomposes as 1 + 3 + 3 + 5 into irreducibles; the other two (inter- 
changed by i) come from 12-dimensional orthogonal representations of A§ which 
decompose as 3 + 4 + 5 into irreducibles. 

(d) If to = 8, then G has exactly one regular 2 35-triple; it has type 
(-D4A3, A 4 A 3 , A\A\)\ it also has exactly one ip G Hom res (A.5, G) up to conjugacy: 
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it comes from a 16-dimensional orthogonal representation of A5 which decomposes 
as 1 + 3 + 3 + 4 + 5 into irreducibles. 

If m = 7 or m > 9 then G has no regular 235-triples; therefore, 

Hom re3 (A,G) = 0. 

We see that in these cases each regular ip comes from a homomorphism A5 — > 

S0 2m . 

1.10. Assume that G = G a d is simple of type E m . From [Fl], [F2],[F3] we see 
that: 

if m = 6, then G has exactly two regular 235-triples (interchanged by t); they 
have type (A 5 A U A%, A 2 A\); 

if m = 7, then G has exactly two regular 235-triples (interchanged by t); they 
have type (A 7 , A 5 A 2 , A 3 A 2 A 1 ); 

if m = 8, then G has exactly one regular 235-triple; it has type (D 8 , Ag, 
From [F1],[F2],[F3] we see also that in each of these cases, a regular 235-triple is 
associated with at least one ip G Hom reg (A5,G). (Another proof of this fact is 
given by Lemma 4.3.) This ip is in fact unique up to conjugacy. (See [F2], [F3] for 
Eq, E 7 and p = 0; see 4.5, 4.6 for the general case.) 

2. Inequalities 

2.1. Let W be a Weyl group and let V be the reflection representation of W (over 
C). Let V = ©j>oVj be the algebra of polynomials V — > C modulo the ideal 
generated by the VF-invariant polynomials of degree > 0; here Vj is the image of 



the space of polynomials of degree j. Now dimV = W and W acts naturally 
on V preserving the grading. Let £{W) be the set of irreducible representations 
of W over C (up to isomorphism). For any E G £(W) let b w (E) (resp. b' w (E)) 
be the smallest (resp. largest) integer j such that Homw(E, Vj) 7^ 0. Note that 
bw(E), b' w (E) are well defined. 

Assume now that W is the Weyl group of G as in 1.2. Let v = v(G). Let 
(C 2 ,C 3 , C 5 ) be a 235-triple for G. Let (g 2 , £3, #5) G C7 2 x C7 3 x C 5 . We denote 
by VF n (resp. by z/ n ) the Weyl group (resp. the number of positive roots) of 
Zoidn): n = 2,3,5. We may regard W 2: W3 : Ws as Weyl subgroups of W. Let 
V^ n ^ = ©jV( n ) be defined in terms of W n in the same way as V = ©j Vj is defined 
in terms of W. For any E G £(W) let b n (E) (resp. b' n (E)) be the smallest (resp. 
largest) integer j such that Hom^ n V^ n ) 7^ 0. Note that b n (E),b' n (E) are 

well defined for n = 2, 3, 5. 

Let sgn be the sign representation of W. We have 

(a) b(E) = v-b'(E® sgn), b n (E) = v n - b' n (E © sgn), n = 2,3,5. 

Now £{W) is a union of equivalence classes called families, see [L, 4.2]. For example 
the unit representation 1 is a family by itself. If T is a family, then {£'©sgn; E G J 7 } 
is again a family. For any family T of W we set 

a(^) = min b(E), a'{F) = m^b'(E), 
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a n (JF) = min b n (E) , a' n (J 7 ) = max b' n {E). 



From (a) we deduce that 



(b) a(jF) = v — d{T ® sgn), a n (jF) = v n — d n {F ® sgn), n = 2, 3, 5. 

Proposition 2.2. In the setup of 2.1, assume that either 

(1) (C^C^Cs) is regular, or 

(2) any simple component of G a d has rank < 8 and is not of type E%. 
Then for any family T we have 

(a) a(F) - En=2,3,5 a "( jr ) ^ r (Gad); 

equivalently (see 2.1(b)), 

(b) En=2,3,5 - <-" + En"n+ r{G ad ) . 

Moreover, if (C2, C3, C5) is not regular, then (a) and (b) are strict inequalities. 
If (C27C37C5) is regular, then (a) is strict for T 7^ {sgn} and (b) is strict for 
T 7^ {1}. If (C2, C3, C5) is regular, then (a) is an equality for T = {sgn} and (b) 
is an equality for T = {1}. 

This can be checked with the aid of a computer. We may assume that G = G a d 
is simple. In the case where G is of type E 8 and (C 2 ,C3,C 5 ) is regular, I have 
used tables in [BL], [A]. If we are not in this case then G has rank < 8 and is not 
of type £^8- Then, instead of [BL], [A], one can use the CHEVIE package [C] to 
get tables for the quantities b n (E) (or rather the analogous quantities where W n is 
replaced by any reflection subgroup of W) . From this information, one can check 
the required inequalities by hand. (I am grateful to M. Geek for his help with the 
CHEVIE package.) 

2.3. We expect that 2.2 continues to hold even if we drop the assumptions (1),(2). 
One can check that 2. 2(a), (b) hold for G of type A without the assumptions 
(1),(2). When (C2, C3, C5) is regular, then 2.2(a) for T = {sgn} is just the equality 

2.4. We illustrate 2.2(a) in the case when G = G a d is of type £^8 an d (C 2 , C3, C5) 
is regular. The subgroups W n of W are as in 1.10. In the following list, there is 
one line for each family T of W, containing the information 

D- a{F) - a 2 (F) - a 3 (F) - a 5 (T) =? 
where D is the degree of the " special" representation in T . Note that T is deter- 
mined by D and a{J-). 

l;0-0-0-0 = 

8; 1-1-1-1 = -2 

35; 2-2-1-0 = -1 

112;3- 0-0-0 = 3 

210;4-0-l-0 = 3 

560; 5-1-1-0 = 3 

567; 6-2-1-1 = 2 



HOMOMORPHISMS OF THE ALTERNATING GROUP A 5 INTO REDUCTIVE GROUPS71 



700; 6 — 


l — u 


— 


= 5 


1 A C\C\ T 

1400; 7 - 


- 1 — 


— 


n — 


r. 



1400; 8- 


-I — 


— 


n 


o 


3240; 9 - 


- 3 — 


1 — 


n — 


r. 
o 


2268; 10 


— A- 


- 2 - 


n - 
- u - 


- A 


2240; 10 


— 3 - 


" 1 - 


n - 
u — 


- U 


4096; 11 


— A - 


- 2 - 


i _ 
i — 


- zL 


525; 12 - 


- A — 


3 — 


i 

i — 


A 


41ZUU 


1 9 
1Z 


— A - 


- 2 - 


n - 

u - 


- u 


2800 


13 


A 

— A - 


- 3 - 


1 _ 

1 — 


- V 


4536 


13 


— 5 - 


- 2 - 


n - 
- u - 


- D 


2835 


14 


— 6 - 


- 2 - 


n - 

u — 


- u 


6075 


14 


— - 


- 3 - 


1 _ 

1 — 


- A 


4200 


15 


— 5 - 


- 3 - 


1 - 

1 — 


- V 


5600 


20 


— 8 - 


- 5 - 


o _ 
z — 


- 


4480 


16 


— 6 - 


- 3 - 


n - 
u — 


_ 7 


5600 


21 


— 8 - 


- 5 - 


o _ 
- Z - 


- D 


4200 


21 


— 9 - 


- 5 - 


9 _ 
- z — 


- 


6075 


22 


— 10 


— 5 


9 

— z 


K 

— o 


2835 


22 


— 10 


— 5 


9 

— Z 


— o 


4536 


23 


— 9 - 


- 5 - 


9 _ 
- z — 




2800 


25 


— 11 


— 7 


Q 

— o 


A 

— ft 


4200 


24 


— 10 


— 6 


9 

— z 


— fi 

— U 


525; 36 - 


- 16 - 


- 9 - 


o — 


- u 


4096; 26 


— 11 


— 7 


Q 

— o 


c 

— o 


2240; 28 


— 12 


— 7 


o 

— o 


R 

— D 


2268; 30 


— 12 


— 8 


/I 

— i 


— V 


3240; 31 


— 13 


o 

— 8 


-4 


= 6 


1400; 32 


1 A 

— 14 


— 9 


- 5 


= 4 


1400; 37 


— 16 


— 9 


- 5 


= 7 


700; 42 - 


- 18- 


- 12 


- 6 


= 6 


567; 46 - 


- 22 - 


- 13 


- 7 


= 4 


560; 47 - 


- 21 - 


- 13 


- 7 


= 6 


210; 52- 


- 24- 


- 16 


-8 


= 4 


112; 63- 


- 28- 


- 18 


- 10 = 


35; 74 - 


34- 


22 - 


- 12 


= 6 


8; 91 


- 43 - 28 - 


16 = 


= 4 


1;120- 


56- 


36- 


-20 


= 8 



3. Fg-STRUCTURES 

3.1. In this section we fix G as in 1.2. We assume that p is a prime > 7 and that 
k is an algebraic closure of the field with p elements. Let q be a power of p and 
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let F q be the subfield of k with q elements. Let F : G — » G be the Frobenius map 
corresponding to an F^-rational structure on G. 

For any integer m > 1 we write m = m p m p > where m p is a power of p and m v > 
is prime to p. If is a map of a set Y into itself, we set = {y G Y; <j>(y) = y}. 

3.2. Let e G = (_i)F 9 -rankof G For any j. e T (G) F and any G Hom(T F ,C*) 
let i?y G : G F — > C be the character of the virtual representation denoted in [DL, 
4.3] by F$p- (We choose a prime number 1,1 ^ p, an algebraic closure (Qz of the 
Z-adic numbers and a field isomorphism Qz = C; then R^, a virtual representation 
over Q;, can be viewed as a virtual representation over C.) Let £(G F ) be the set 
of irreducible (complex) characters of G F . Let Si(G F ) be the set of irreducible 
unipotent characters of G F that is, the irreducible characters p of G F such that 
(p : Rj, q)q 7^ for some T G T F . Here (:) G is the standard hermitian inner 
product of class functions G F — » C. Let 0{G F ) be the set of all class functions 
4> : G F — ► C such that (0 : R e TyG ) G = for any T G T{G) F and any G 
Hom(T F ,C*). 

Lemma 3.3. Let p G £\{G F ). There exists a unique G F -invariant function f p : 
T(G) F — > C and a unique £ G 0(G F ) such that 



G 



E 



f p {T)R) r G + £. 



TeT(G) 1 



This follows immediately from the definition of £i(G F ) and from the orthogo- 
nality relations [DL, 6.8]. 

Lemma 3.4. Let g e G F be semisimple. Let T G T(G) F , G Hom(T F , C*). We 
/lave 



E <%')• 



9'eC G F( 3 )nT 



See [DL, 7.2]. 

3.5. For any G F -invariant function / : T(G) F — ► C let 

-l 



Rf,G = 



G 



E 

TeT(G) F 



f(T)R?r,G- 



If if is a connected reductive F-stable subgroup of G with r(H) = r{G) and / is 
as above, we shall write R/,h instead of Rf\ T ( H)F ,H- 



Lemma 3.6. In the setup of 3.5 we have YliTeT{G) F e G^rf{T) 



G 1 



Rf jG (l) may be computed using the formula in [DL, 7.1] for R^ G (l); the lemma 
follows. 
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3.7. We denote by Gf (resp. Gf) the group of all 9 G Hom(G F , C*) such that 
°\k{G)f = 1 ( res P- d L(Gn = !)• Clearly, Gf C Gf . 

If T G T(G) F and i? G Hom(T F , C*), let S(T, tf) be the subgroup of G defined 
as in [DL, 5.19]. If if is an F-stable connected reductive subgroup of G with 
r(if) = r(G), let if F be the set of all 9 G iff such that for some (or equivalently, 
any) T G T(if) F we have S(T, 6\ t f) = H. Let iff be the set of all 9 G iff such 
that = 9' 9" for some 9' E H F , 9" G iff. 

Let Ag,f be the set of all pairs (if, 0) where if is an F-stable connected re- 
ductive subgroup of G with r(if ) = r(G) and G if£. Let F be the set of all 
F-stable connected reductive subgroups if of G such that if F ^ 0. 

Lemma 3.8. Assume that Z G = Z G . Let if G X' GF . Then any orbit of the 



obvious Ng(H) f -action on H F has cardinal N G (H) F / H 



Let 9 G ff£. Let S e be the stabilizer of 9 in N G (H) F . We show that S e = H F . 
The inclusion if F C is immediate. Now let g G Sq. We must show that 
g G if F . Let T G T(if) F be maximally split. Since gHg -1 = if, we see that 
gTg~ x G T(if) F is maximally split hence gTg~ x = hTh~ x for some ft G if F . 
Replacing g by gh~ x we see that we may assume that gTg~ x = T. Now conjugation 
by g keeps fixed 9\ t f. By [DL, 5.13], g is a product of elements in Nh(T). In 
particular, g G if. Hence # G if F . The lemma is proved. 

3.9. Let (H,9) G X g ,f- Let 8 H ,e( G ) be the set of a11 P G £(G F ) such that 
(p : R%T G ) G ^ for some T G T(if) F . (We write T instead of 0| t f.) According 
to [DL, 6.3, 5.20], if Zq = Z G , we have a partition 

(a) S(G F ) = U (Hj0) S H , (G F ) 

where (if, 9) runs over a set of representatives for the G F -orbits on X G p. 

According to [L], if Z G = Z G , for any (if, 9) G Xg,f, there exists k G {1, —1} 
and a bijection 

(b) £ 1 (H F ) ^ S H ,e(G F ) 

(denoted by p <-> p H ' e ) such that for any p G £i(H F ) and any T G T(if) F we have 



(p : Rt,h)h = K(p 



HA 



t>9t 



7° 
Z G 



Lemma 3.10. Assume that Zq - 
Let p G £i(if F ). Let /' : T(if) F 

3/ p (T)^ H + i, £ G 0(H F ) (see 3.3) 



Let (if , 0) G Xq,f- Let k be as above. 
C 6e £/ie if F -invariant function such that 



H 1 



E 



TeT(H) F 



H 1 



E 



(a,) VFe /lave p^'* 9 

f&j Let g G G F 6e semisimple. We have 



TeT{H) F 



f p (T)R T F G +£ t ' with ^' G 0(G F ). 



p H ' e (g) = KZ G (g) F ,J2 e z c(9) e zo H{9 ') Z H {g') F f R fP ^ {gl) (l)9(g') 
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where g' runs through a set of representatives for the conjugacy classes in H F that 
are contained in C g f (g) . 

(c) We have p H > (l) = k 



G 1 



H 1 



y CGCHRfP,H(l)- 

To prove the identity in (a) it is enough to show that both sides have the same 
inner product with Rj^,' G for any T' G T(H) F . By [DL, 6.8], the inner product of 

the right hand side with Rpi' G is 

(d) k V ^21^(91 e G F - gi Tg^ = T', Ad( 9l ) carries 9 T to T >) 

TeT(H) F | ti 



For any g\ in (d) we see that Ad(gi) carries S(T, 6>t) to S(T', $t') hence it carries 
{H, 9) to (H, 9). By the argument in the proof of 3.8 we have g\ G H F . Thus, (d) 
is equal to 

' E f P (TM9ieH F - 9l Tg^=T'). 

TeT(H) F 



K 



H 1 



Now 



(p H > 9 : R d T r G ) G = n(p : B^,„) H 



= K 



H 1 



E /TOeff^Tr 1 ^'). 

TeT(H) F 



This proves (a). We prove (b). By [DL, 7.5] we have £'(flO = 0. Hence 

^]r(T)R e T - G (g) 



P H > d (g) = « 



E 

TeT(H) F 



= K 



H 1 



E nT)e z%{g) e T \ZG{gy 

TeT(H) F 



E 



g 'eC GF ( s )nT 



K 



H 1 



E e ^(j7) e ^(9') E ^ P ( T ) 

3 'eC GF ( 3 )n// TeT(H) F ;g'eT 



x e ^( 3 ') eT 



Ip' 



H 1 



E ^( 9 )^(,0 | L fl /,zo 0,0(1) 1 Z G (g) F \ 9(g') 

g'eC GF (g)nH 



and (b) follows. Now (c) is a special case of (b). The lemma is proved. 
The following result is well known. 
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Lemma 3.11. Let A,B,C be three conjugacy classes in G F and let (a, 6, c) G 
A x B x C. We have 



(a) tf{(a,&,c) G A x B x C\abc = 1} 



A 


B 


c\ 




G F 





p{a)p{b)p{c) 



3.12. If A, £?, C are three semisimple conjugacy classes in G F ', we set 

£?G F-A B C = Yl Q- iy(Z G(«))-^( Z &( & ))-^(^W)+^(G)-r(G ad ) 

pe£i(G^) 

X RfP,Z%{a)X)RfP,Z% (6)( 1 )%P,Z0(c)( 1 )^/^,G(l)" 1 

where / p : T(G) F — > C are defined as in 3.3 and (a, 6, c) G A x B x C. 

If, in addition, H G F and A', C" are three semisimple conjugacy classes 
in H F such that A' ci,B'c S, C" C C, we set 

tG,F;H;A,B,C;A',B',C = e Z%{a') e Z%{a') e Z%{b') e Z%{b') e Z%{c') e Z° H {c') 



X 







C 




v < 


z G {by 


i>> 


z G (cy 

¥ 


1 


H F 


v < 








Z H {a') F 


p' 


Z H (b') F 


P > 


Zh{c') F 


- P > 


G F 


V' 



x q -6+u(Z%(a'))+v(Z H (b'))+v(Z° H (c'))-v(H)+r(H) 

where (a', 6', c ') G A' x £?' x C" and 

5 = 3u(G) - i/(Zg(a)) - u(Z G (b)) - u(Z G (c)). 
Note that e z o (a) = e z o (a/) , e z o (&) = e z o (&/) , e z o (c) = e z o (c/) . We set 

^G,F;H;A,B,C;A',B',G' 

= tG,F;H;A,B,C;A',B',C"QH,F;A',B',C"q~ dlmZH ^ 6(a'b'c'), 

den:? 



Xg,F;H;A,B,C = ^ Yg,F;H;A,B,C;A' ,B' ,G' , 



A',B',C' 



where A' (resp. C") runs through the conjugacy classes in H F that are con- 
tained in A (resp. B,C). 

Let F be a set of representatives for the orbits of the natural G F -action on 



^g,f- 



12 



G. LUSZTIG 



Lemma 3.13. Assume that Zq = Z G . We have 



q- s #{(a,~b,c)eAxBxC;abc=l} = £ X g,F;h ; a,b,c 



HeX G,F 



NdHf/H 1 



We rewrite the right hand side of 3.11(a) using the partition 3.9(a), the bijection 
3.9(b) and using Lemma 3.8; we obtain 



Jt{(o,6,c) G A x B x C;abc = 1} = N^H^/H 1 



X' 



G,F;H;A,B,C 



where for any H G X' G F we set 



X'g,F;H;A,B,C 



A 


B 


c\ 




G F 





E E 



p H > e (a)p H ' e (b)p H > 9 (c) 



We evaluate p H ' e (a) , p H > 9 (b) , p H ' e (c) using 3.10(b), and p H ' e (l) using 3.10(c); we 
obtain X' GF . H . ABC = q 6 Xc,f-,h-a,b,c (see 3.12). The lemma follows. 

Lemma 3.14. (a) In the setup of 3.12 we have \tG,F-H:A,B,c-.A',B' ,c\ < c where 
c > is an integer depending only on the Coxeter graph on G (and not on 
q,F,H,A,B,C,A',B',C). 
(b) More precisely, 



ez« {a>)t Z % (a') e Z« {b>)tZ% (f)Czg, (C) e Z£ (<) < 



where c' > is an integer depending only on the Coxeter graph on G (and not on 
q,F,H,A,B,C,A',B',C'). 

This is easily checked. 

Lemma 3.15. Assume that Zq = Z G . 

(a) There exists an integer c > depending only on the Coxeter graph of G 

(and not on q,F) such that 



X'g,f 



< c. 



(b) There exists an integer c' > depending only on the Coxeter graph of G 



( and not onq, F) such that for any H G X' G F we have 



Let G* be a connected reductive group of type dual to that of G. Let F' : G* — > 
G* be the Frobenius map corresponding to an F 9 -rational structure on G. Let 
yc*,F' be the set of all pairs (E, g) where g G G* F is semisimple and E = Zq* (g). 
Let y' G * pi be the set of all subgroups E of G* such that (E, g) G !Vg*,f' for some 
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g. For E E y G * F > let Z E ^ = {g E Z E ;Z G *(g) = E}. Let Z E ,v be the union of 
all connected components of Z E that contain some point in Z E ^. Let y' G * jE ' be 
a set of representatives for the orbits of the natural G* F '-action on y G * F ,. 

Using [DL, 5.20, 5.24] we can reduce the statements of the lemma to statements 
about G*: 

(a') There exists an integer c > depending only on the Coxeter graph of G* 
(and not on q, F') such that 



G*F' 



ryF' 7 F' 



< 



< c. 

(b') There exists an integer c' > depending only on the Coxeter graph of G 
(and not on q,F') such that for any (E,g) E y' G * p> we have 

c /gdim(Z°)-l_ 

The proof of (a') is standard. For (t/) we note that any connected component of 
Z E p is an irreducible variety isomorphic to Z E and its intersection with Z E ^ is 
open and dense in it. The lemma is proved. 

3.16. Let H be an T-stable connected reductive subgroup of G such that r(H) = 
r(G). Let W be the Weyl group of G and let W be the Weyl group of H. Note 
that F acts naturally on W and on W . 

Let T , B be a pair consisting of an T-stable maximal torus of G and an in- 
stable Borel subgroup of G containing To. Let Tq,S be a pair consisting of an 
T-stable maximal torus of H and an T-stable Borel subgroup of H containing Tq. 
We identify W = N G (T )/T , W = N H (T{ j )/T ' in the standard way. We choose 
7 E G such that 7T 7 _1 = Tq. Let m = r y~ 1 F(-f). We have m E N G (T ). We have 
an imbedding W C W induced by N H (T^) — > N G (T ), h i-> 7 _1 /i7- We identify 
W with its image under this imbedding. The map F : W — ► W corresponds 
under this identification to the restriction of the map F' : W W induced by 
iV G (T ) -> N G (T ), g ^ mF{g)m-\ 

Let E be an irreducible representation of W with a given isomorphism X E : 
E — > E of finite order such that \ E (we) = F(w)X E (e) for all w E W, e E E. 

Define f E : T{G) F — * C by f E (T) = tr(w T X E : E E) where w T is the image 
in W of z~ 1 F(z) E No (To) where z E G is such that zTqz' 1 = T. (Note that 
/s(T) is independent of the choice of z.) 

Define V'- in terms of W in the same way as Vj was defined in 2.1 in terms of W . 
Then W acts naturally on V'- and there is a natural isomorphism Xj : — > V'- 
of finite order such that \j(w'x) = F'(w')Xj(x) for all w' E W, x E Vj. 

We define a linear map Xj : Romw'(E, Vj) — > B.omw'(E, V'j) by (Aj(£))(e) = 

X~ 1 ^(mX E (e)) for all £ E Rom w > (E, V^), e EE. (One checks easily that A,(£) E 

HomvK^-E 1 , V'-).) One also checks that Xj is a map of finite order. With this 
notation, we have 

(a) Rf E ,H(l) = ^2 tr(Aj, Homw (E, Vj))q j . 

j>0 
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This follows in a standard way from the definitions using the formula [DL, 7.1] for 
r t,g(X)- 

Lemma 3.17. Let H G X' GF . Let g G H F be a semisimple element. Assume that 
Zjj(g) is split over F q and that F acts trivially on Zjj(g). Let g' G H F be such 
that g,g' are conjugate in H . Then: 

(a) ez°( 9 >) = e z%( 9 '); 

(b) F acts trivially on Zn{g')- 

We can find a maximal torus T of Z Q H {g) such that F{T) = T and T is F g -split. 
Let T be a maximal torus of Z^(g') such that FT' = T'. We can find x G H 
such that xgx~ x = g'^xTx -1 = T'. Let m = x~ 1 F(x). Then m G Zjj(g) and 
m G N H (T). Let n G ,TZ H (resp. K' G ,K' H ) be the set of roots of Z G (g),Z%(g) 
(resp. Z G (g'), Zjj(g')) with respect to T (resp. T"). Then F acts naturally on 
1Z g ,1Zh,'R' G7 71' h . Let 7t' G + ,7l' H + be a set of positive roots for 1Z' G ,1Z' H respec- 
tively. It is well known that 

e T ,e z o (50 = (_l)»(a^' G +;F(a)e^-^ G +) ) 

e T ,e z l {gl) = (-i)K^n' H +;F( a) en' H -n' H +)_ 

Now Ad(x) establishes bijections IZq — T^'g anc ^ — T^'h under which the 
action of F on VJ Gl VJ H corresponds to the action given by Ad(m) on 1Z g ,1Zh 
(since F acts trivially on 1Z G ,1Z H ). Also under this bijection, TZ' G + (resp. TZ' H + ) 
corresponds to a set of positive roots 1Z G (resp. 7l~jj) for TZq (resp. It 
follows that 

e T >Zz° G {g>) = (-l)«( ae ^ + ;Ad(m)(a)e^ G -^ G + ) j 

er , ez o ( = (_l)lt(aeKif + ;Ad(n>)(a)eR H -R H + ) i 

The right hand sides of the previous two equalities may be also interpreted as the 
determinant of the linear map induced by Ad(m) on the group of characters of 
T. (We use that m defines an element in the Weyl group of Z G (g) or Z Q H {g) with 
respect to T.) Hence those right hand sides coincide. It follows that (a) holds. 

Now Ad(x) also induces an isomorphism Zn{g) — Zii(g') under which the 
action of F on Zn(g') corresponds to the action of Ad(m) on Zn(g) (since F acts 
trivially on Zn{g))- Hence to prove (b) it is enough to show that Ad(m) acts 
trivially on Zjj(g). Since m G Zjj(g), this follows from the well known fact that 
Zn{g) is commutative. The lemma is proved. 

3.18. In this subsection we assume that (H,F) is like (G,F) in 3.1. We assume 
that H = Hder and that C is a semisimple conjugacy class in H. Let a G C. We 
assume that F(a) = a and that F acts trivially on Zn{a). We assume also that 
F acts trivially onZ = Z^. 

(a) Define \i : Z — > H F /ir(H F ) by z i— > ir(h) where h G H satisfies h~ 1 F(h) = 
z. Then n is a group isomorphism. 
The proof is standard. 

Let Z 1 be the set of all z G Z such that a, za are conjugate in H; here a G 
n~ 1 (a) C H. (This definition does not depend on the choice of a.) Note that Z' 
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is a subgroup of Z. Let X be the set of if F -conjugacy classes contained in C F . 

(b) Define // : Z' — » X by z i— > C h f (Tv(g)a7f(g)~ 1 ) where g & H is such that 
g~ 1 F(g) = gi.giag^ 1 = za. Then /i' is a bisection. 

The proof is standard. 

(c) The composition Z' Z — > H F /n(H F ) (the first map is the inclusion) 

coincides with the composition Z' — > X — » H F /n(H F ) (the second map is A i— > 

7r(g) where g E A). 

This follows from the definitions. 

(d) Assume that N > 1 is an integer such that a N = 1. XTien z N = 1 /or aZZ 
z E Z'. 

Indeed if z E Z' we have = gag -1 for some g E G. Taking A^-th powers gives 
z N a N = ga N g~ x hence (using a N E Z): z N a N = gg -1 ^ = a N , hence z N = 1. 

Assume now that (C' 2 , C3, C5) is a 235-triple for i7. Let a n G C^,n = 2,3,5. 
We assume that F(a n ) = a n and F acts trivially on Z#(a n ) for n = 2, 3, 5. 

Let Z^, X n be defined like Z',X in terms of C^, a n instead of C, a. 

(e) TTie map X2 x AT3 x X5 — > H F /ir(H F ), given by (A,B,C) 1— > 7r(a6c) where 
(a, 6, c) G A x £? x C, is injective. 

Using (a),(b),(c), we see that this is equivalent to the statement that the map 
Z' 2 x Z3 x Z5 — > Z given by multiplication in Z is injective. From (d) we see that 
any element z E Z' n satisfies z n = 1. Then the injectivity of the map above follows 
from the fact that an element of a finite abelian group can be written in at most 
one way as a product of elements of order dividing 2, 3, 5. 

In the remainder of this subsection we assume in addition that H E X' G F where 
(G, F) as in 3.1 is such that G = G a d, G simple of type E m , m = 6, 7, 8. (Recall 

that H = Hder-) 

(f) The image I of the map in (e) is exactly U = {x E H F /tt(H f ); x 30 = 1}. 



By (d), we have I C U. Now [T] = Z' 2 Z 3 Z5 and | U \ can be easily determined 



by inspection of the various cases. We find that [T] = | U \ hence I = U. 

((f) ought to be true also in type D, but we have not checked it, as we don't 
need it.) In particular, the map in (e) is a bijection except if (G, H) is of type 
(E$, D 5 x A 3 ) or (£7, A 3 x A 3 x Ai) in which case the image has index 2. 

Lemma 3.19. Assume that G = G a d- Let F : G G be the Frobenius map 
corresponding to an F p -rational structure on G. There exists an integer sq > 1 
such that if s > 1 is an integer divisible by so, and H E X' G F s is semisimple, then 
H is split over F pS . 

Let G* be as in the proof of 3.15. Let Fq : G* — > G* be the Frobenius map 
corresponding to an F p -rational structure on G*. As in the proof of 3.15, we see 
that it is enough to verify the following statement: 

(a) There exists an integer sq > 1 such that if s > 1 is an integer divisible by 
so, and g E G* F " is a semisimple element such that H = Z G *(g) is semisimple, 
then H is split over F pS . 
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The set of all g' E G* such that g' is semisimple and Zq* (g') is semisimple is a 
union of finitely many conjugacy classes Ai, A2, ■ ■ ■ , A m in G*. Pick g r E A r for 
r G [1, m]. Let _£f r = Z^* (g r ). We can find an integer s > 1 such that g r E G* F ° S ° 
and H r is split over F p s , r = 1, . . . , m. If s > 1 is an integer divisible by sq then 
g r E G* F ° and H r is split over F pS , r = 1, . . . , m. Now let g E G* F ° be such 

that iif = Zc*(g) is semisimple. We have g E A r for some r. Hence j G i r ° • 
Now ZQ*(g r ) is connected since G* is simply connected. It follows that G* F acts 

transitively (by conjugation) on A r ° . In particular, G,<7 r are conjugate under 
G* F s . It follows that H = Z G *(g) is split over Fps . The lemma is proved. 

4. Estimates 

Lemma 4.1. Let G, o, F 6e as in 3.1. Let A, £?, C be conjugacy classes in G F such 
that A C C2, B C C3, Cc C5 where (C2, C3, C5) a 235-triple in G. Assume that 
the assumptions of 2.2 hold. Then \Gg,f-a,b,c\ < c where c is a constant de- 
pending only on G (not on F, o, A, B, C). Moreover, if (C2, C3, C5) is not regular, 
then \Qg f-a b c\ < c'g -1 where c' is a constant depending only on G (not on 
F,q,A,B,C). ' 

We use the notation of 2.1. We may assume that (g2, #3, #5) E Ax B x C. Now 
F acts naturally on W. Hence it acts naturally on the set of families of W. By 
[L, 4.23] we have a partition 

S l (G F ) = U T S 1 AG F ) 
[T runs over the F-stable families in W) such that the following holds: if p E 

£!,AG F ), then 

E 

where E runs through the representations in T that are "F-stable", are 
complex numbers that are bounded above (when q varies) and : T(G) F — » C 
is as in 3.16. Using 3.16(a) we see that 

RfM 1 ) = ^2s p ,E^2tT(<T j ,liom w (E,V j ))^, 

E j 

%^z G ( 5 Jo(l) = ^ Sp , s ^tr(^^Hom w?l (£;| Wrl ,vf ) )y, n = 2,3,5, 
b i 

where o~j, are linear maps of finite order. Then 

R f p ,ZG(92) ( 1 ) R fP,Z G (g 3 r(i)RfP,Z G (g 5 y > ( 1 ) 

is a linear combination with bounded coefficients of powers o J where 

j < b' 2 (E) + b' 3 (E') + b' 5 (E") with E, E', E" E T 
(hence j < J2 n =2 3 5 a 'ni.^'))i while .R/p 5 g(1) is a linear combination with bounded 
coefficients of powers g J where j < b'(E) with E E T (hence j < a'(jF)) and in 
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fact it is known that the coefficient of q a '^ is non-zero and its inverse is bounded 
above. Now 

QG,F;A,B,C = J2fGg,F;A,B,C,F 

where T runs over the B-stable families in W and Gg,F;A,b,c,f is defined like 
Gg,f-,a,b,c but with p restricted to £\^{G F ). The arguments above show, us- 
ing 2.2(b), that each Qg,f-,a,b,c,t is bounded above; and, in the case where 
{pi-, C3, C5) is not regular, that each qGG,F-,A,B,c,F is bounded above. The lemma 
follows. 

4.2. In the setup of 4.1, assume that (C2, 63,65) is regular. By the method of 
4.1, we see that 

(a) q(GG,F-.A,B,c — 1) is bounded above. 

Lemma 4.3. Let G be as in 3.1. Let (C2, C3, C5) be a regular 235-triple in Gder- 
Then there exists ifj G Hom(^4 5 , G) such that tp gives rise to (C2, C3, C5) as in 1.7. 

Since this statement depends only on Gder and we can find an imbedding of 
Gder info a connected reductive group with connected centre and derived subgroup 
Gder: w e see that we may assume that G has connected centre. We may also assume 
that dim G > and that the lemma is true when G is replaced by a group whose 
derived group has strictly smaller dimension than that of Gder- Assume that there 
is no ip as in the lemma (for G). Let (#2, <?3, 95) £ C2 x C3 x C5. Since g n G G^er, 
we may choose g, B as in 3.1 in such a way that g n G 7i"((j ) for n = 2, 3, 5. We 
may also assume that G, Z G (g 2 ), Z G (g 3 ), Z G (g 5 ) are split over F q and that q is 
large. 

We write the identity in 3.13 for (A, B, C) = {C G F(g 2 ),C G F(g 3 ),C G F(g 5 )). The 
left hand side of that identity is 0, by our assumption. We deduce that 

-1 



(a) N g{H) F /H 



Xg,F;H;A,B,C = 0. 



We show that 

(b) qXG,F-H-,A,B,c is bounded above (when q, F vary) for any B G X G F , H ^ G. 
With notation as in 3.12, we have qX Gj F-H-,A,B,c = where are 

given by 

^ tG,F;H;A,B,C;A',B',C'GH,F;A',B',C'QQ~ dlmZH ^ 6(a'b'c'); 
(A>,B>,C>) 

in (resp. X]")' 0^') ^") runs over all triples of conjugacy classes in H F such 
that A' C A,B' C B, C" C C and A' C I', B' C B', C' C C" where (A', B', C") is 
a 235-triple in B, and (A', B', C') is regular (resp. non-regular) for B. 

Now Y^," is bounded above; indeed, in each term, t Gj F-H-A,B,c-,A' ,B' ,c is bounded| 
above (by 3.14), qGh,f-A',b',C" is bounded above (by 4.1) and 

q~ dim z h Y.eeHF O(a'b'c') is bounded above since q~ dim z * H F 



is bounded above. 
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We show that Yl' is bounded above. In each term, tG,F-H-A,B,C;A\B',C' is 
bounded above (by 3.14), Gh,f-A' ,B' ,c is bounded above (by 4.1) and it is enough 
to show that qq~ dimZ n J2 0eHF O(a'b'd) is bounded above or that 



qq- d ™ Z » 



J2 6(a'b'c')-qq- dimZ " 



is bounded above. Now qq dimZ " J2 deH F_ H F9(a'b'd) is bounded above since 

qq~ dimZ H — is bounded above (by 3.15(b)). Hence it is enough to show 
that J2 0eH F O(a'b'c') = 0. Since H F is a union of iff-cosets in H F , it is enough 

to show that a'b'c' £ Hd er - Assume that a'b'c' G Hder for some (hence any) 
(a',b',d) G A' x B' x C. Since a', 6', d have orders dividing 2, 3, 5, it follows 
that each of a',b',d is in Hder- Hence A',B',C are contained in Hder- Since 
(A', B', C') is regular in H, we may apply to it the induction hypothesis; we see 
that there exists ip G Hom(^.5,iy) such that tp(x2) G A',ip(x3) G B', ipfes) G C . 
Since A' C C 2 ,B' C C 3 , C' C C 5 , we see that ip(x n ) G C n for n = 2,3,5. This 
contradicts our assumption. Thus, we have a'b'd £ Hder and the boundedness of 
^2' (hence (b)) is established. 

Using now (a), we deduce that qXG,F-,G;A,B,c is bounded above. Setting t = 

tG,F;G;A,B,C;A,B,C,G = Qg,F;A,B,C, We have 



qX G , F ;G;A,B,C = qtQq-*™ Z e J2 0(92939$) = qtQq- dimZ o 



G 



eeGl 



since g-zdzds G ir(G F ). From 3.14(b) we see that t 1 is bounded above and it is 

-l 

is bounded above. It follows that qQ is bounded above, 
'he lemma is proved. 



G 



clear that q d[mZ G 



This contradicts 4.2. 



4.4. Assume now that G (as in 1.2) is adjoint of type Eg and that (C2, C3, C5) is 
the unique regular 235 triple of G. Let d be the number of G-orbits on 
Hom res (^4 5 , G). To determine d we may assume that k is as in 3.1. Let g, F be 
as in 3.1. We write the identity 3.11(a) for (A,B,C) = (C[,C[,C[) (these are 
three conjugacy classes of G F ). Assume that we can evaluate the right hand side 
of 3.11(a) for infinitely many q and that it is of the form Aq D + lower powers of q 
where A is a constant. Then 3.11(a) implies that {(a, 6, c) E C2XC3X C 5 ; abc = 1} 
has exactly A irreducible components of dimension D and it follows that d = A. 
(This strategy was suggested in [S].) 

Theorem 4.5. Let G be as in 1.2 and let (C2, C3, C5) be a regular 235-triple of 
G. Assume that G — G a a- Let N = 



Let g n G C n and let N n = Z G (g n ) 
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n = 2,3,5. Then, up to G-conjugacy, there are exactly N2N:iNF elements if) G 
Hom res (^4.5 , G) that give rise to (C2, 63,65). 

We may assume that G is simple. In the case where G is of type A or D, the 
theorem can be obtained from the results in §1. In the rest of the proof we assume 
that G is of type E m ,m = 6, 7, 8 (although the same proof should work without 
this assumption). 

We will carry out the strategy in 4.4, using the results in §2, §3. We may assume 
that k is as in 3.1. Let Fq be a Frobenius map for an F p -rational structure on G. 
Let q = p s , F = Fq where s' is sufficiently large and divisible by a fixed integer 
s' Q > 1. By choosing s' Q apropriately we may assume that F acts trivially on Zq 
that g n G tt(G f ), that ZQ(g n ) is split over F q and that F acts trivially on Zc(g n ) 
for n = 2, 3, 5. (The choice of s' will be made more precise later in the proof.) Let 
5 = 3u — u 2 — vj, — z/5. (Notation of 2.1.) Let A = N2 j^ N ■ We will show that 

(a) q(q~ 6 t{(92,93,95) e C( x C F x C F ;g 2 g 3 g 5 = 1} - A) 

is bounded above when q varies or equivalently, that 

${(92,93,95) e A x B x C;g 2 g 3 g 5 = 1} - A) 

(A,B,C)eX 

is bounded above. Here X is the set of triples (A, B, C) of conjugacy classes in 
G F such that A C C 2 , B C C 3 , C C C 5 . By 3.13, it is enough to show that 

( b ) 9(Z)(A,s,c)eA'^G,F;G;^,B,c - A) is bounded above, 
( c ) 9E(a,b,c)6A-^g,F;H;A,b,c is bounded above for any H G X' G F ,H ^ G. 
We have 

52(A,B,C)£X X G,F;G;A,B,C = Ea,B,C tG,F;G;A,B,C;A,B,cGG,F;A,B,C EgeG^ 

where (a, 6, c) G A x £? x C. The sum over 6 is unless a&c G n(G F ) (or equiva- 
lently, each of a, 6, c is in rc(G F )), in which case it is N. Thus, (b) is equivalent to 
the statement that 

q(tG,F-G;A,B,C;A,B,cGG,F-A,B,cN - A) is bounded above; 
here (A, B,C) G X is uniquely determined by the condition 

A C n(G F ),B C tt((5 f ),C C tt(G f ). 
This follows from the fact that q(tG,F-,G;A,B,c-,A,B,c — (N 2 N 3 N 5 ) _1 ) is bounded 
above (see 3.14, 3.17) and q(GG,F;A,B,c ~~ 1) is bounded above (see 4.2). Thus, (b) 
holds. 

Now, in (c), we have (as in the proof of 4.3) 

qY.{A,B,C)eX X G,F;H;A,B,C = E +E 

where E'> E" are gi ven by 



i G,F;H-A,B,C;A',B',C'QH,F-A',B',C'qq dlraZ x ^ O(a'b'c'); 
(A',B',C) 9eHl 
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in Yl\ (resp. X)")> (A',B',C) runs over the set A" (resp. Af") consisting of all 
triples of conjugacy classes in H F such that A' C A',B' C B',C C C" where 
(i', is a 235-triple in if, I' C C 2 ,S' C C 3 ,C" C C 5 and (i',B',C") 

is regular (resp. non-regular) for H; moreover (A, B, C) is the unique triple of 
conjugacy classes in G F such that A' C A, B' C B, C C C. 

Now Yl" is bounded above (exactly as in the proof of 4.3). 

We now estimate Yl' ■ 

Assume first that dim Zjj > 0. Again, Y' is bounded above: in each term, 
tG,F-H;A,B,C;A',B',C' is bounded above (by 3.14), Gh,f-A',b',c is bounded above 
(by 4.1) and it is enough to show that qq~ dimZ w YeeH F (a'b'c.) is bounded above 
or that 

qq - d imZ° H 9{p , Vd) _ qq - d imZ° H YeeHF-HF ^'b'c') 

is bounded above. Now qq~ dimZ H YeeH F -H F d(a'b'c') is bounded above since 



qq 



dimZ 



is bounded above (by 3.15(b)). Hence it is enough to show 
that YoeHg (a'b'c') = 0. Since H F is a union of H F -cosets in H F , it is enough 

to show that a'b'c' ^ Hder- Assume that a'b'c' G Hder for some (hence any) 
(a',b',c') G A' x B' x C Since a', 6', d have orders dividing 2, 3, 5, it follows 
that each of a',b',c' is in Hder- Hence A',B',C are contained in Hder- Since 
(A', B' ', C') is regular in H, we may apply to it Lemma 4.3; we see that there 
exists ip G Hom(^. 5 ,i7) such that •i/ , ( ;r 2) £ -4', ^(^3) £ -B', ^(^5) e Since 
A' C C^-B' C C 3 ,C" C C 3 , we see that V(^n) G C n for n = 2,3,5. Since 
(C2, 63,65) is regular for 6, we have tp G Hom res ( v 4.5, G). This contradicts the fact 
that Zq(iI)(A§)) contains the non-trivial torus Z^. Thus, we have a'b'c' Hd er 
and the boundedness of Y 1S established. 

Next, assume that dimZ H = 0. We show that Y is bounded above. Since for 
(A',B',C) G A", q(GH,F;A',B',c - 1) is bounded above (see 4.2) and 
tG,F-H;A,B,C;A',B',c is bounded above (see 3.14), it is enough to show that, for 
any regular 235-triple (63,63,65) of H such that C' n C 6 n , n = 2,3,5, and for 
any 9 G H F , 

Y,(A' ,B' ,C')ey at G,F;H;A,B,C;A' ,B' ,C'0(a'b'c') 

is bounded above. Here y = {(A',B',C) G X';A' C C' 2 ,B' C 6 3 , C C 6 5 } and 
(a', 6', c') G A' x £?' x C". Using 3.14(c) we see that it is enough to show that, for 
any 9 G H F , 

E(A' ,b> ,c)ey T A' ,b> ,c0(a'b'c') = 
where 

tZ% (a')£ Z ° H {a>)£ Z % {V)tZ% {b')^Z% (c') e Z% (c') 
TA',B',C = 



Z H (a') F Z H {b'Y Z H (c') 



>\F 



J\F 



By choosing s' appropriately, we may assume that H is split over F q (see 3.19), 
that F acts trivially on Z^, that C' n contains an F g -rational point h n such that 
Zjj(h n ) is split over F q and F acts trivially on Zn(h n ), n = 2,3,5. Using 3.17, 
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we see then that, in the last fraction, the numerator is 1 and the denominator 
is Zuih'i) Zuihz) Zn{h§) ; in particular, ta',b',c is independent of A',B',C. 



Thus, we are reduced to showing that 

E(A',B',c)ey d ( a ' b ' c ') =0 
Using the results at the end of 3.18, we see that the last equality is equivalent to 

the equality 

Y,xEH F /tt(H f );x 30 =1 ®( X ) = 

where 9 is regarded as a character of H F /tt(H f ). More precisely, if (G, H) is not of 
type (E 8 ,D 5 xA 3 ) or (E 7 , A 3 x A 3 x A x ) , we must show that E^/^) Q( x ) = 
and this follows from the fact that 9^1 (since 9 G H F ). If (G,H) is of type 
(E 8: D 5 x A 3 ) or (E 7 ,A 3 x A 3 x A{), we must show that the restriction of 9 to 
the subgroup of H F /n(H F ) Z/4Z or ^ Z/4Z © Z/2Z) consisting of all x such 
that x 2 = 1 is 7^ 1; this again follows from 9 G H F . 
This proves (c). Thus, (a) is proved. 

Let X = {(a, 6, c) G C2 x C 3 x C5; a&c = 1} (an algebraic variety defined over F q ). 



From (a) we see that there exists an integer M > such that q s (q Ss — A) < 



-F s 



M for s = 1, 2, As it is known, this implies that among the irreducible 

components of X there are exactly A of maximum dimension, and that maximum 
dimension is 5. By 1.3, any G-orbit on X has dimension 6. It follows that X 
has exactly A G-orbits. Since X is naturally in bijection with the set of all if) G 
Hom(^45, G) that give rise to (C2, C3, C5) we see that the theorem is proved. 

4.6. For G = G ad of type E 6 ,E 7 ,E 8 , the fraction N2 ^ 3 n 5 is = 2^1 = 
1, y-j— j- = 1 respectively. We see that in type E 8 there is exactly one ifj G 
Hom reff (^45, G) up to G-conjugacy. 

For G = G ad of type A x , A 2l A 3l A 4l A 5 , the fraction N2 fi 3Ns is 533 = 1, 
Th = 1 ^2h = 2 ^Th = 1 ^2h = 1 respectively. 

For G = G ad of type D A , D 5 , D 6 , D 8 , the fraction j^ff^ is = 1, = 2 ' 
4TT = TTi = 1 respectively. 



5. Relation to homomorphisms PGL 2 (k) — > G 

5.1. In this section we assume that either p = or p is large enough. Let G be as 
in 1.2. Let $ : PGL2(k) — >Gbea homomorphism. Composing $ with one of the 
two regular homomorphisms A5 — > PGL2(k) (up to PGL2(/c)-conjugacy, see 1.8), 
we obtain two homomorphisms A5 — > G (which may or may not be G-conjugate). 

In [S], it is pointed out that by applying this construction to a $ whose image 
contains a regular unipotent element in G (of type E m ) we obtain a regular ip. 
Applying the same procedure for G = G ad of type A m _i, we see that all regular 
ip in 1.8 can be thus obtained for m = 2, 3, 5, 6 but only one of the two regular tfj 
is thus obtained for m = 4. This procedure can be applied for G as in 1.9. In that 
case, for m = 4, the first two ip in 1.9(a) are obtained; for m = 5, the first ip in 
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1.9(b) is obtained; for m = 6, the first ip in 1.9(c) is obtained; for m = 8, the first 
V> in 1.9(d) is obtained. 

Let us analyze what happens if we use some other 

Let G be as in 1.9 and assume that $ is such that its image contains a subregular 
unipotent element of G. For m = 4, we thus obtain the third and fourth ip in 1.9(a); 
for m = 5, we thus obtain the second ip in 1.9(b); for m = 6, we thus obtain the 
second and third ip in 1.9(c). 

Let G be as in 1.9(d). If $ is such that its image contains a unipotent element 
whose Jordan blocks in the standard representation have sizes 1,3,5,7, then the 
associated ip is regular. 

If G (as in 1.10) is of type E$ and $ is such that the centralizer of its image 
is the symmetric group in 5 letters, then the ip obtained from $ by the procedure 
above is again regular (the corresponding 235-triple is the one described in 1.10). 
Using 4.5, the ifj thus obtained must be the same up to conjugacy as the one 
attached to a regular unipotent class. 

These arguments together with 4.5 show that, if G = G a a is simple, then any 
ip G Hom res („4.5, G) is obtained from some $ : PGL 2 (k) — + G, except for one case 
in type A 3 . 
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